{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "lesbian-springer",
   "metadata": {},
   "source": [
    "## Seldon Core 2 Pipeline Examples\n",
    "\n",
    "This notebook illustrates a series of Pipelines showing of different ways of combining flows of data and conditional logic. We assume you have Seldon Core 2 running locally.\n",
    "\n",
    "### Models Used\n",
    "\n",
    " * `gs://seldon-models/triton/simple` an example Triton tensorflow model that takes 2 inputs INPUT0 and INPUT1 and adds them to produce OUTPUT0 and also subtracts INPUT1 from INPUT0 to produce OUTPUT1. See [here](https://github.com/triton-inference-server/server/tree/main/docs/examples/model_repository/simple) for the original source code and license.\n",
    " * Other models can be found at https://github.com/SeldonIO/triton-python-examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "running-antarctica",
   "metadata": {},
   "source": [
    "### Model Chaining\n",
    "\n",
    "Chain the output of one model into the next. Also shows chaning the tensor names via `tensorMap` to conform to the expected input tensor names of the second model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0f2ab2a3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: tfsimple1\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/triton/simple\"\n",
      "  requirements:\n",
      "  - tensorflow\n",
      "  memory: 100Ki\n",
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: tfsimple2\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/triton/simple\"\n",
      "  requirements:\n",
      "  - tensorflow\n",
      "  memory: 100Ki\n"
     ]
    }
   ],
   "source": [
    "!cat ./models/tfsimple1.yaml\n",
    "!cat ./models/tfsimple2.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f9e073d7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model load -f ./models/tfsimple1.yaml \n",
    "!seldon model load -f ./models/tfsimple2.yaml "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "997b4028",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model status tfsimple1 -w ModelAvailable | jq -M .\n",
    "!seldon model status tfsimple2 -w ModelAvailable | jq -M ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beb58fda",
   "metadata": {},
   "source": [
    "The pipeline below chains the output of `tfsimple1` into `tfsimple2`. As these models have compatible shape and data type this can be done. However, the output tensor names from `tfsimple1` need to be renamed to match the input tensor names for `tfsimple2`. We do this with the `tensorMap` feature.\n",
    "\n",
    "The output of the Pipeline is the output from `tfsimple2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3d017ede",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Pipeline\r\n",
      "metadata:\r\n",
      "  name: tfsimples\r\n",
      "spec:\r\n",
      "  steps:\r\n",
      "    - name: tfsimple1\r\n",
      "    - name: tfsimple2\r\n",
      "      inputs:\r\n",
      "      - tfsimple1\r\n",
      "      tensorMap:\r\n",
      "        tfsimple1.outputs.OUTPUT0: INPUT0\r\n",
      "        tfsimple1.outputs.OUTPUT1: INPUT1\r\n",
      "  output:\r\n",
      "    steps:\r\n",
      "    - tfsimple2\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./pipelines/tfsimples.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "following-winning",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline load -f ./pipelines/tfsimples.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "artistic-kentucky",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"pipelineName\": \"tfsimples\",\r\n",
      "  \"versions\": [\r\n",
      "    {\r\n",
      "      \"pipeline\": {\r\n",
      "        \"name\": \"tfsimples\",\r\n",
      "        \"uid\": \"ciep26qi8ufs73flaiqg\",\r\n",
      "        \"version\": 2,\r\n",
      "        \"steps\": [\r\n",
      "          {\r\n",
      "            \"name\": \"tfsimple1\"\r\n",
      "          },\r\n",
      "          {\r\n",
      "            \"name\": \"tfsimple2\",\r\n",
      "            \"inputs\": [\r\n",
      "              \"tfsimple1.outputs\"\r\n",
      "            ],\r\n",
      "            \"tensorMap\": {\r\n",
      "              \"tfsimple1.outputs.OUTPUT0\": \"INPUT0\",\r\n",
      "              \"tfsimple1.outputs.OUTPUT1\": \"INPUT1\"\r\n",
      "            }\r\n",
      "          }\r\n",
      "        ],\r\n",
      "        \"output\": {\r\n",
      "          \"steps\": [\r\n",
      "            \"tfsimple2.outputs\"\r\n",
      "          ]\r\n",
      "        },\r\n",
      "        \"kubernetesMeta\": {}\r\n",
      "      },\r\n",
      "      \"state\": {\r\n",
      "        \"pipelineVersion\": 2,\r\n",
      "        \"status\": \"PipelineReady\",\r\n",
      "        \"reason\": \"created pipeline\",\r\n",
      "        \"lastChangeTimestamp\": \"2023-06-29T14:11:40.101677847Z\",\r\n",
      "        \"modelsReady\": true\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline status tfsimples -w PipelineReady | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "87f10a5c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"model_name\": \"\",\r\n",
      "  \"outputs\": [\r\n",
      "    {\r\n",
      "      \"data\": [\r\n",
      "        2,\r\n",
      "        4,\r\n",
      "        6,\r\n",
      "        8,\r\n",
      "        10,\r\n",
      "        12,\r\n",
      "        14,\r\n",
      "        16,\r\n",
      "        18,\r\n",
      "        20,\r\n",
      "        22,\r\n",
      "        24,\r\n",
      "        26,\r\n",
      "        28,\r\n",
      "        30,\r\n",
      "        32\r\n",
      "      ],\r\n",
      "      \"name\": \"OUTPUT0\",\r\n",
      "      \"shape\": [\r\n",
      "        1,\r\n",
      "        16\r\n",
      "      ],\r\n",
      "      \"datatype\": \"INT32\"\r\n",
      "    },\r\n",
      "    {\r\n",
      "      \"data\": [\r\n",
      "        2,\r\n",
      "        4,\r\n",
      "        6,\r\n",
      "        8,\r\n",
      "        10,\r\n",
      "        12,\r\n",
      "        14,\r\n",
      "        16,\r\n",
      "        18,\r\n",
      "        20,\r\n",
      "        22,\r\n",
      "        24,\r\n",
      "        26,\r\n",
      "        28,\r\n",
      "        30,\r\n",
      "        32\r\n",
      "      ],\r\n",
      "      \"name\": \"OUTPUT1\",\r\n",
      "      \"shape\": [\r\n",
      "        1,\r\n",
      "        16\r\n",
      "      ],\r\n",
      "      \"datatype\": \"INT32\"\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer tfsimples \\\n",
    "    '{\"inputs\":[{\"name\":\"INPUT0\",\"data\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],\"datatype\":\"INT32\",\"shape\":[1,16]},{\"name\":\"INPUT1\",\"data\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],\"datatype\":\"INT32\",\"shape\":[1,16]}]}' | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5c33133e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"outputs\": [\r\n",
      "    {\r\n",
      "      \"name\": \"OUTPUT0\",\r\n",
      "      \"datatype\": \"INT32\",\r\n",
      "      \"shape\": [\r\n",
      "        \"1\",\r\n",
      "        \"16\"\r\n",
      "      ],\r\n",
      "      \"contents\": {\r\n",
      "        \"intContents\": [\r\n",
      "          2,\r\n",
      "          4,\r\n",
      "          6,\r\n",
      "          8,\r\n",
      "          10,\r\n",
      "          12,\r\n",
      "          14,\r\n",
      "          16,\r\n",
      "          18,\r\n",
      "          20,\r\n",
      "          22,\r\n",
      "          24,\r\n",
      "          26,\r\n",
      "          28,\r\n",
      "          30,\r\n",
      "          32\r\n",
      "        ]\r\n",
      "      }\r\n",
      "    },\r\n",
      "    {\r\n",
      "      \"name\": \"OUTPUT1\",\r\n",
      "      \"datatype\": \"INT32\",\r\n",
      "      \"shape\": [\r\n",
      "        \"1\",\r\n",
      "        \"16\"\r\n",
      "      ],\r\n",
      "      \"contents\": {\r\n",
      "        \"intContents\": [\r\n",
      "          2,\r\n",
      "          4,\r\n",
      "          6,\r\n",
      "          8,\r\n",
      "          10,\r\n",
      "          12,\r\n",
      "          14,\r\n",
      "          16,\r\n",
      "          18,\r\n",
      "          20,\r\n",
      "          22,\r\n",
      "          24,\r\n",
      "          26,\r\n",
      "          28,\r\n",
      "          30,\r\n",
      "          32\r\n",
      "        ]\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer tfsimples --inference-mode grpc \\\n",
    "    '{\"model_name\":\"simple\",\"inputs\":[{\"name\":\"INPUT0\",\"contents\":{\"int_contents\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]},\"datatype\":\"INT32\",\"shape\":[1,16]},{\"name\":\"INPUT1\",\"contents\":{\"int_contents\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]},\"datatype\":\"INT32\",\"shape\":[1,16]}]}' | jq -M ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9878e97",
   "metadata": {},
   "source": [
    "We use the Seldon CLI `pipeline inspect` feature to look at the data for all steps of the pipeline for the last data item passed through the pipeline (the default). This can be useful for debugging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "126144a5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldon.default.model.tfsimple1.inputs\tciep298fh5ss73dpdir0\t{\"inputs\":[{\"name\":\"INPUT0\", \"datatype\":\"INT32\", \"shape\":[\"1\", \"16\"], \"contents\":{\"intContents\":[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]}}, {\"name\":\"INPUT1\", \"datatype\":\"INT32\", \"shape\":[\"1\", \"16\"], \"contents\":{\"intContents\":[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]}}]}\r\n",
      "seldon.default.model.tfsimple1.outputs\tciep298fh5ss73dpdir0\t{\"modelName\":\"tfsimple1_1\", \"modelVersion\":\"1\", \"outputs\":[{\"name\":\"OUTPUT0\", \"datatype\":\"INT32\", \"shape\":[\"1\", \"16\"], \"contents\":{\"intContents\":[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32]}}, {\"name\":\"OUTPUT1\", \"datatype\":\"INT32\", \"shape\":[\"1\", \"16\"], \"contents\":{\"intContents\":[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}}]}\r\n",
      "seldon.default.model.tfsimple2.inputs\tciep298fh5ss73dpdir0\t{\"inputs\":[{\"name\":\"INPUT0\", \"datatype\":\"INT32\", \"shape\":[\"1\", \"16\"], \"contents\":{\"intContents\":[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32]}}, {\"name\":\"INPUT1\", \"datatype\":\"INT32\", \"shape\":[\"1\", \"16\"], \"contents\":{\"intContents\":[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}}], \"rawInputContents\":[\"AgAAAAQAAAAGAAAACAAAAAoAAAAMAAAADgAAABAAAAASAAAAFAAAABYAAAAYAAAAGgAAABwAAAAeAAAAIAAAAA==\", \"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==\"]}\r\n",
      "seldon.default.model.tfsimple2.outputs\tciep298fh5ss73dpdir0\t{\"modelName\":\"tfsimple2_1\", \"modelVersion\":\"1\", \"outputs\":[{\"name\":\"OUTPUT0\", \"datatype\":\"INT32\", \"shape\":[\"1\", \"16\"], \"contents\":{\"intContents\":[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32]}}, {\"name\":\"OUTPUT1\", \"datatype\":\"INT32\", \"shape\":[\"1\", \"16\"], \"contents\":{\"intContents\":[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32]}}]}\r\n",
      "seldon.default.pipeline.tfsimples.inputs\tciep298fh5ss73dpdir0\t{\"modelName\":\"tfsimples\", \"inputs\":[{\"name\":\"INPUT0\", \"datatype\":\"INT32\", \"shape\":[\"1\", \"16\"], \"contents\":{\"intContents\":[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]}}, {\"name\":\"INPUT1\", \"datatype\":\"INT32\", \"shape\":[\"1\", \"16\"], \"contents\":{\"intContents\":[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]}}]}\r\n",
      "seldon.default.pipeline.tfsimples.outputs\tciep298fh5ss73dpdir0\t{\"outputs\":[{\"name\":\"OUTPUT0\", \"datatype\":\"INT32\", \"shape\":[\"1\", \"16\"], \"contents\":{\"intContents\":[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32]}}, {\"name\":\"OUTPUT1\", \"datatype\":\"INT32\", \"shape\":[\"1\", \"16\"], \"contents\":{\"intContents\":[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32]}}]}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline inspect tfsimples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf445e9b",
   "metadata": {},
   "source": [
    "Next, we look get the output as json and use the `jq` tool to get just one value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "10b5bca0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"inputs\": [\r\n",
      "    {\r\n",
      "      \"name\": \"INPUT0\",\r\n",
      "      \"datatype\": \"INT32\",\r\n",
      "      \"shape\": [\r\n",
      "        \"1\",\r\n",
      "        \"16\"\r\n",
      "      ],\r\n",
      "      \"contents\": {\r\n",
      "        \"intContents\": [\r\n",
      "          1,\r\n",
      "          2,\r\n",
      "          3,\r\n",
      "          4,\r\n",
      "          5,\r\n",
      "          6,\r\n",
      "          7,\r\n",
      "          8,\r\n",
      "          9,\r\n",
      "          10,\r\n",
      "          11,\r\n",
      "          12,\r\n",
      "          13,\r\n",
      "          14,\r\n",
      "          15,\r\n",
      "          16\r\n",
      "        ]\r\n",
      "      }\r\n",
      "    },\r\n",
      "    {\r\n",
      "      \"name\": \"INPUT1\",\r\n",
      "      \"datatype\": \"INT32\",\r\n",
      "      \"shape\": [\r\n",
      "        \"1\",\r\n",
      "        \"16\"\r\n",
      "      ],\r\n",
      "      \"contents\": {\r\n",
      "        \"intContents\": [\r\n",
      "          1,\r\n",
      "          2,\r\n",
      "          3,\r\n",
      "          4,\r\n",
      "          5,\r\n",
      "          6,\r\n",
      "          7,\r\n",
      "          8,\r\n",
      "          9,\r\n",
      "          10,\r\n",
      "          11,\r\n",
      "          12,\r\n",
      "          13,\r\n",
      "          14,\r\n",
      "          15,\r\n",
      "          16\r\n",
      "        ]\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline inspect tfsimples --format json | jq -M .topics[0].msgs[0].value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "dimensional-hours",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline unload tfsimples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "outside-inspiration",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon model unload tfsimple1\n",
    "!seldon model unload tfsimple2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c6a8e27",
   "metadata": {},
   "source": [
    "### Model Chaining from inputs\n",
    "\n",
    "Chain the output of one model into the next. Shows using the input and outputs and combining."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "7cbf98a0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: tfsimple1\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/triton/simple\"\n",
      "  requirements:\n",
      "  - tensorflow\n",
      "  memory: 100Ki\n",
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: tfsimple2\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/triton/simple\"\n",
      "  requirements:\n",
      "  - tensorflow\n",
      "  memory: 100Ki\n"
     ]
    }
   ],
   "source": [
    "!cat ./models/tfsimple1.yaml\n",
    "!cat ./models/tfsimple2.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f0674da8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model load -f ./models/tfsimple1.yaml \n",
    "!seldon model load -f ./models/tfsimple2.yaml "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "53ab8c9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model status tfsimple1 -w ModelAvailable | jq -M .\n",
    "!seldon model status tfsimple2 -w ModelAvailable | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "aa3c0927",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Pipeline\r\n",
      "metadata:\r\n",
      "  name: tfsimples-input\r\n",
      "spec:\r\n",
      "  steps:\r\n",
      "    - name: tfsimple1\r\n",
      "    - name: tfsimple2\r\n",
      "      inputs:\r\n",
      "      - tfsimple1.inputs.INPUT0\r\n",
      "      - tfsimple1.outputs.OUTPUT1\r\n",
      "      tensorMap:\r\n",
      "        tfsimple1.outputs.OUTPUT1: INPUT1\r\n",
      "  output:\r\n",
      "    steps:\r\n",
      "    - tfsimple2\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./pipelines/tfsimples-input.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5c0aa3a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline load -f ./pipelines/tfsimples-input.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "0fc44852",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"pipelineName\": \"tfsimples-input\",\r\n",
      "  \"versions\": [\r\n",
      "    {\r\n",
      "      \"pipeline\": {\r\n",
      "        \"name\": \"tfsimples-input\",\r\n",
      "        \"uid\": \"ciep2fii8ufs73flair0\",\r\n",
      "        \"version\": 1,\r\n",
      "        \"steps\": [\r\n",
      "          {\r\n",
      "            \"name\": \"tfsimple1\"\r\n",
      "          },\r\n",
      "          {\r\n",
      "            \"name\": \"tfsimple2\",\r\n",
      "            \"inputs\": [\r\n",
      "              \"tfsimple1.inputs.INPUT0\",\r\n",
      "              \"tfsimple1.outputs.OUTPUT1\"\r\n",
      "            ],\r\n",
      "            \"tensorMap\": {\r\n",
      "              \"tfsimple1.outputs.OUTPUT1\": \"INPUT1\"\r\n",
      "            }\r\n",
      "          }\r\n",
      "        ],\r\n",
      "        \"output\": {\r\n",
      "          \"steps\": [\r\n",
      "            \"tfsimple2.outputs\"\r\n",
      "          ]\r\n",
      "        },\r\n",
      "        \"kubernetesMeta\": {}\r\n",
      "      },\r\n",
      "      \"state\": {\r\n",
      "        \"pipelineVersion\": 1,\r\n",
      "        \"status\": \"PipelineReady\",\r\n",
      "        \"reason\": \"created pipeline\",\r\n",
      "        \"lastChangeTimestamp\": \"2023-06-29T14:12:14.711416101Z\",\r\n",
      "        \"modelsReady\": true\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline status tfsimples-input -w PipelineReady | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "904cf65a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"model_name\": \"\",\r\n",
      "  \"outputs\": [\r\n",
      "    {\r\n",
      "      \"data\": [\r\n",
      "        1,\r\n",
      "        2,\r\n",
      "        3,\r\n",
      "        4,\r\n",
      "        5,\r\n",
      "        6,\r\n",
      "        7,\r\n",
      "        8,\r\n",
      "        9,\r\n",
      "        10,\r\n",
      "        11,\r\n",
      "        12,\r\n",
      "        13,\r\n",
      "        14,\r\n",
      "        15,\r\n",
      "        16\r\n",
      "      ],\r\n",
      "      \"name\": \"OUTPUT0\",\r\n",
      "      \"shape\": [\r\n",
      "        1,\r\n",
      "        16\r\n",
      "      ],\r\n",
      "      \"datatype\": \"INT32\"\r\n",
      "    },\r\n",
      "    {\r\n",
      "      \"data\": [\r\n",
      "        1,\r\n",
      "        2,\r\n",
      "        3,\r\n",
      "        4,\r\n",
      "        5,\r\n",
      "        6,\r\n",
      "        7,\r\n",
      "        8,\r\n",
      "        9,\r\n",
      "        10,\r\n",
      "        11,\r\n",
      "        12,\r\n",
      "        13,\r\n",
      "        14,\r\n",
      "        15,\r\n",
      "        16\r\n",
      "      ],\r\n",
      "      \"name\": \"OUTPUT1\",\r\n",
      "      \"shape\": [\r\n",
      "        1,\r\n",
      "        16\r\n",
      "      ],\r\n",
      "      \"datatype\": \"INT32\"\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer tfsimples-input \\\n",
    "    '{\"inputs\":[{\"name\":\"INPUT0\",\"data\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],\"datatype\":\"INT32\",\"shape\":[1,16]},{\"name\":\"INPUT1\",\"data\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],\"datatype\":\"INT32\",\"shape\":[1,16]}]}' | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "58a2f2ef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"outputs\": [\r\n",
      "    {\r\n",
      "      \"name\": \"OUTPUT0\",\r\n",
      "      \"datatype\": \"INT32\",\r\n",
      "      \"shape\": [\r\n",
      "        \"1\",\r\n",
      "        \"16\"\r\n",
      "      ],\r\n",
      "      \"contents\": {\r\n",
      "        \"intContents\": [\r\n",
      "          1,\r\n",
      "          2,\r\n",
      "          3,\r\n",
      "          4,\r\n",
      "          5,\r\n",
      "          6,\r\n",
      "          7,\r\n",
      "          8,\r\n",
      "          9,\r\n",
      "          10,\r\n",
      "          11,\r\n",
      "          12,\r\n",
      "          13,\r\n",
      "          14,\r\n",
      "          15,\r\n",
      "          16\r\n",
      "        ]\r\n",
      "      }\r\n",
      "    },\r\n",
      "    {\r\n",
      "      \"name\": \"OUTPUT1\",\r\n",
      "      \"datatype\": \"INT32\",\r\n",
      "      \"shape\": [\r\n",
      "        \"1\",\r\n",
      "        \"16\"\r\n",
      "      ],\r\n",
      "      \"contents\": {\r\n",
      "        \"intContents\": [\r\n",
      "          1,\r\n",
      "          2,\r\n",
      "          3,\r\n",
      "          4,\r\n",
      "          5,\r\n",
      "          6,\r\n",
      "          7,\r\n",
      "          8,\r\n",
      "          9,\r\n",
      "          10,\r\n",
      "          11,\r\n",
      "          12,\r\n",
      "          13,\r\n",
      "          14,\r\n",
      "          15,\r\n",
      "          16\r\n",
      "        ]\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer tfsimples-input --inference-mode grpc \\\n",
    "    '{\"model_name\":\"simple\",\"inputs\":[{\"name\":\"INPUT0\",\"contents\":{\"int_contents\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]},\"datatype\":\"INT32\",\"shape\":[1,16]},{\"name\":\"INPUT1\",\"contents\":{\"int_contents\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]},\"datatype\":\"INT32\",\"shape\":[1,16]}]}' | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "1cd53dd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline unload tfsimples-input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "91cce2e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon model unload tfsimple1\n",
    "!seldon model unload tfsimple2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fca716ed",
   "metadata": {},
   "source": [
    "### Model Join\n",
    "\n",
    "Join two flows of data from two models as input to a third model. This shows how individual flows of data can be combined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "5e41768d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: tfsimple1\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/triton/simple\"\n",
      "  requirements:\n",
      "  - tensorflow\n",
      "  memory: 100Ki\n",
      "---\n",
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: tfsimple2\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/triton/simple\"\n",
      "  requirements:\n",
      "  - tensorflow\n",
      "  memory: 100Ki\n",
      "---\n",
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: tfsimple3\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/triton/simple\"\n",
      "  requirements:\n",
      "  - tensorflow\n",
      "  memory: 100Ki\n"
     ]
    }
   ],
   "source": [
    "!cat ./models/tfsimple1.yaml\n",
    "!echo \"---\"\n",
    "!cat ./models/tfsimple2.yaml\n",
    "!echo \"---\"\n",
    "!cat ./models/tfsimple3.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "fcbc23cc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model load -f ./models/tfsimple1.yaml \n",
    "!seldon model load -f ./models/tfsimple2.yaml \n",
    "!seldon model load -f ./models/tfsimple3.yaml "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "f8bb2efb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model status tfsimple1 -w ModelAvailable | jq -M .\n",
    "!seldon model status tfsimple2 -w ModelAvailable | jq -M .\n",
    "!seldon model status tfsimple3 -w ModelAvailable | jq -M ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "145b0851",
   "metadata": {},
   "source": [
    "In the pipeline below for the input to `tfsimple3` we join 1 output tensor each from the two previous models `tfsimple1` and `tfsimple2`. We need to use the `tensorMap` feature to rename each output tensor to one of the expected input tensors for the `tfsimple3` model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "cd58ecba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Pipeline\r\n",
      "metadata:\r\n",
      "  name: join\r\n",
      "spec:\r\n",
      "  steps:\r\n",
      "    - name: tfsimple1\r\n",
      "    - name: tfsimple2\r\n",
      "    - name: tfsimple3      \r\n",
      "      inputs:\r\n",
      "      - tfsimple1.outputs.OUTPUT0\r\n",
      "      - tfsimple2.outputs.OUTPUT1\r\n",
      "      tensorMap:\r\n",
      "        tfsimple1.outputs.OUTPUT0: INPUT0\r\n",
      "        tfsimple2.outputs.OUTPUT1: INPUT1\r\n",
      "  output:\r\n",
      "    steps:\r\n",
      "    - tfsimple3\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./pipelines/tfsimples-join.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "ce9e163b",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline load -f ./pipelines/tfsimples-join.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "7593c77f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"pipelineName\": \"join\",\r\n",
      "  \"versions\": [\r\n",
      "    {\r\n",
      "      \"pipeline\": {\r\n",
      "        \"name\": \"join\",\r\n",
      "        \"uid\": \"ciep2k2i8ufs73flairg\",\r\n",
      "        \"version\": 1,\r\n",
      "        \"steps\": [\r\n",
      "          {\r\n",
      "            \"name\": \"tfsimple1\"\r\n",
      "          },\r\n",
      "          {\r\n",
      "            \"name\": \"tfsimple2\"\r\n",
      "          },\r\n",
      "          {\r\n",
      "            \"name\": \"tfsimple3\",\r\n",
      "            \"inputs\": [\r\n",
      "              \"tfsimple1.outputs.OUTPUT0\",\r\n",
      "              \"tfsimple2.outputs.OUTPUT1\"\r\n",
      "            ],\r\n",
      "            \"tensorMap\": {\r\n",
      "              \"tfsimple1.outputs.OUTPUT0\": \"INPUT0\",\r\n",
      "              \"tfsimple2.outputs.OUTPUT1\": \"INPUT1\"\r\n",
      "            }\r\n",
      "          }\r\n",
      "        ],\r\n",
      "        \"output\": {\r\n",
      "          \"steps\": [\r\n",
      "            \"tfsimple3.outputs\"\r\n",
      "          ]\r\n",
      "        },\r\n",
      "        \"kubernetesMeta\": {}\r\n",
      "      },\r\n",
      "      \"state\": {\r\n",
      "        \"pipelineVersion\": 1,\r\n",
      "        \"status\": \"PipelineReady\",\r\n",
      "        \"reason\": \"created pipeline\",\r\n",
      "        \"lastChangeTimestamp\": \"2023-06-29T14:12:32.938603415Z\",\r\n",
      "        \"modelsReady\": true\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline status join -w PipelineReady | jq -M ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f82fe9c0",
   "metadata": {},
   "source": [
    "The outputs are the sequence \"2,4,6...\" which conforms to the logic of this model (addition and subtraction) when fed the output of the first two models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "2151ee4c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"outputs\": [\r\n",
      "    {\r\n",
      "      \"name\": \"OUTPUT0\",\r\n",
      "      \"datatype\": \"INT32\",\r\n",
      "      \"shape\": [\r\n",
      "        \"1\",\r\n",
      "        \"16\"\r\n",
      "      ],\r\n",
      "      \"contents\": {\r\n",
      "        \"intContents\": [\r\n",
      "          2,\r\n",
      "          4,\r\n",
      "          6,\r\n",
      "          8,\r\n",
      "          10,\r\n",
      "          12,\r\n",
      "          14,\r\n",
      "          16,\r\n",
      "          18,\r\n",
      "          20,\r\n",
      "          22,\r\n",
      "          24,\r\n",
      "          26,\r\n",
      "          28,\r\n",
      "          30,\r\n",
      "          32\r\n",
      "        ]\r\n",
      "      }\r\n",
      "    },\r\n",
      "    {\r\n",
      "      \"name\": \"OUTPUT1\",\r\n",
      "      \"datatype\": \"INT32\",\r\n",
      "      \"shape\": [\r\n",
      "        \"1\",\r\n",
      "        \"16\"\r\n",
      "      ],\r\n",
      "      \"contents\": {\r\n",
      "        \"intContents\": [\r\n",
      "          2,\r\n",
      "          4,\r\n",
      "          6,\r\n",
      "          8,\r\n",
      "          10,\r\n",
      "          12,\r\n",
      "          14,\r\n",
      "          16,\r\n",
      "          18,\r\n",
      "          20,\r\n",
      "          22,\r\n",
      "          24,\r\n",
      "          26,\r\n",
      "          28,\r\n",
      "          30,\r\n",
      "          32\r\n",
      "        ]\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer join --inference-mode grpc \\\n",
    "    '{\"model_name\":\"simple\",\"inputs\":[{\"name\":\"INPUT0\",\"contents\":{\"int_contents\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]},\"datatype\":\"INT32\",\"shape\":[1,16]},{\"name\":\"INPUT1\",\"contents\":{\"int_contents\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]},\"datatype\":\"INT32\",\"shape\":[1,16]}]}' | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "06575cca",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline unload join"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "8bd56a9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon model unload tfsimple1\n",
    "!seldon model unload tfsimple2\n",
    "!seldon model unload tfsimple3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6666e04",
   "metadata": {},
   "source": [
    "### Conditional\n",
    "\n",
    "Shows conditional data flows - one of two models is run based on output tensors from first. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "e3a2c7f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: conditional\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/scv2/samples/triton_23-03/conditional\"\n",
      "  requirements:\n",
      "  - triton\n",
      "  - python\n",
      "---\n",
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: add10\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/scv2/samples/triton_23-03/add10\"\n",
      "  requirements:\n",
      "  - triton\n",
      "  - python\n",
      "---\n",
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: mul10\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/scv2/samples/triton_23-03/mul10\"\n",
      "  requirements:\n",
      "  - triton\n",
      "  - python\n"
     ]
    }
   ],
   "source": [
    "!cat ./models/conditional.yaml\n",
    "!echo \"---\"\n",
    "!cat ./models/add10.yaml\n",
    "!echo \"---\"\n",
    "!cat ./models/mul10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "2104f9bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model load -f ./models/conditional.yaml \n",
    "!seldon model load -f ./models/add10.yaml \n",
    "!seldon model load -f ./models/mul10.yaml "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "7af88e3a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model status conditional -w ModelAvailable | jq -M .\n",
    "!seldon model status add10 -w ModelAvailable | jq -M .\n",
    "!seldon model status mul10 -w ModelAvailable | jq -M ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00e827fb",
   "metadata": {},
   "source": [
    "Here we assume the `conditional` model can output two tensors OUTPUT0 and OUTPUT1 but only outputs the former if the CHOICE input tensor is set to 0 otherwise it outputs tensor OUTPUT1. By this means only one of the two downstream models will receive data and run. The `output` steps does an `any` join from both models and whichever data appears first will be sent as output to pipeline. As in this case only 1 of the two models `add10` and `mul10` runs we will receive their output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "f5bbe84d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Pipeline\r\n",
      "metadata:\r\n",
      "  name: tfsimple-conditional\r\n",
      "spec:\r\n",
      "  steps:\r\n",
      "  - name: conditional\r\n",
      "  - name: mul10\r\n",
      "    inputs:\r\n",
      "    - conditional.outputs.OUTPUT0\r\n",
      "    tensorMap:\r\n",
      "      conditional.outputs.OUTPUT0: INPUT\r\n",
      "  - name: add10\r\n",
      "    inputs:\r\n",
      "    - conditional.outputs.OUTPUT1\r\n",
      "    tensorMap:\r\n",
      "      conditional.outputs.OUTPUT1: INPUT\r\n",
      "  output:\r\n",
      "    steps:\r\n",
      "    - mul10\r\n",
      "    - add10\r\n",
      "    stepsJoin: any\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./pipelines/conditional.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "829f9b18",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline load -f ./pipelines/conditional.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "ae202d26",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"pipelineName\": \"tfsimple-conditional\",\r\n",
      "  \"versions\": [\r\n",
      "    {\r\n",
      "      \"pipeline\": {\r\n",
      "        \"name\": \"tfsimple-conditional\",\r\n",
      "        \"uid\": \"ciepga2i8ufs73flais0\",\r\n",
      "        \"version\": 1,\r\n",
      "        \"steps\": [\r\n",
      "          {\r\n",
      "            \"name\": \"add10\",\r\n",
      "            \"inputs\": [\r\n",
      "              \"conditional.outputs.OUTPUT1\"\r\n",
      "            ],\r\n",
      "            \"tensorMap\": {\r\n",
      "              \"conditional.outputs.OUTPUT1\": \"INPUT\"\r\n",
      "            }\r\n",
      "          },\r\n",
      "          {\r\n",
      "            \"name\": \"conditional\"\r\n",
      "          },\r\n",
      "          {\r\n",
      "            \"name\": \"mul10\",\r\n",
      "            \"inputs\": [\r\n",
      "              \"conditional.outputs.OUTPUT0\"\r\n",
      "            ],\r\n",
      "            \"tensorMap\": {\r\n",
      "              \"conditional.outputs.OUTPUT0\": \"INPUT\"\r\n",
      "            }\r\n",
      "          }\r\n",
      "        ],\r\n",
      "        \"output\": {\r\n",
      "          \"steps\": [\r\n",
      "            \"mul10.outputs\",\r\n",
      "            \"add10.outputs\"\r\n",
      "          ],\r\n",
      "          \"stepsJoin\": \"ANY\"\r\n",
      "        },\r\n",
      "        \"kubernetesMeta\": {}\r\n",
      "      },\r\n",
      "      \"state\": {\r\n",
      "        \"pipelineVersion\": 1,\r\n",
      "        \"status\": \"PipelineReady\",\r\n",
      "        \"reason\": \"created pipeline\",\r\n",
      "        \"lastChangeTimestamp\": \"2023-06-29T14:41:45.133142725Z\",\r\n",
      "        \"modelsReady\": true\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline status tfsimple-conditional -w PipelineReady | jq -M ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a98583af",
   "metadata": {},
   "source": [
    "The `mul10` model will run as the CHOICE tensor is set to 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "0d03d5fe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"outputs\": [\r\n",
      "    {\r\n",
      "      \"name\": \"OUTPUT\",\r\n",
      "      \"datatype\": \"FP32\",\r\n",
      "      \"shape\": [\r\n",
      "        \"4\"\r\n",
      "      ],\r\n",
      "      \"contents\": {\r\n",
      "        \"fp32Contents\": [\r\n",
      "          10,\r\n",
      "          20,\r\n",
      "          30,\r\n",
      "          40\r\n",
      "        ]\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer tfsimple-conditional --inference-mode grpc \\\n",
    " '{\"model_name\":\"conditional\",\"inputs\":[{\"name\":\"CHOICE\",\"contents\":{\"int_contents\":[0]},\"datatype\":\"INT32\",\"shape\":[1]},{\"name\":\"INPUT0\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]},{\"name\":\"INPUT1\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' | jq -M ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8aab3a9c",
   "metadata": {},
   "source": [
    "The `add10` model will run as the CHOICE tensor is not set to zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "def4e8e1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"outputs\": [\r\n",
      "    {\r\n",
      "      \"name\": \"OUTPUT\",\r\n",
      "      \"datatype\": \"FP32\",\r\n",
      "      \"shape\": [\r\n",
      "        \"4\"\r\n",
      "      ],\r\n",
      "      \"contents\": {\r\n",
      "        \"fp32Contents\": [\r\n",
      "          11,\r\n",
      "          12,\r\n",
      "          13,\r\n",
      "          14\r\n",
      "        ]\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer tfsimple-conditional --inference-mode grpc \\\n",
    " '{\"model_name\":\"conditional\",\"inputs\":[{\"name\":\"CHOICE\",\"contents\":{\"int_contents\":[1]},\"datatype\":\"INT32\",\"shape\":[1]},{\"name\":\"INPUT0\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]},{\"name\":\"INPUT1\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "3042fe42",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline unload tfsimple-conditional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "3bda8710",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon model unload conditional\n",
    "!seldon model unload add10\n",
    "!seldon model unload mul10"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ada5634",
   "metadata": {},
   "source": [
    "### Pipeline Input Tensors\n",
    "Access to indivudal tensors in pipeline inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "baa2cb5f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: mul10\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/scv2/samples/triton_23-03/mul10\"\n",
      "  requirements:\n",
      "  - triton\n",
      "  - python\n",
      "---\n",
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: add10\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/scv2/samples/triton_23-03/add10\"\n",
      "  requirements:\n",
      "  - triton\n",
      "  - python\n"
     ]
    }
   ],
   "source": [
    "!cat ./models/mul10.yaml\n",
    "!echo \"---\"\n",
    "!cat ./models/add10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "33692146",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model load -f ./models/mul10.yaml \n",
    "!seldon model load -f ./models/add10.yaml "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "e0493925",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model status mul10 -w ModelAvailable | jq -M .\n",
    "!seldon model status add10 -w ModelAvailable | jq -M ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c5bb324",
   "metadata": {},
   "source": [
    "This pipeline shows how we can access pipeline inputs INPUT0 and INPUT1 from different steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "c8d27ff3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Pipeline\r\n",
      "metadata:\r\n",
      "  name: pipeline-inputs\r\n",
      "spec:\r\n",
      "  steps:\r\n",
      "  - name: mul10\r\n",
      "    inputs:\r\n",
      "    - pipeline-inputs.inputs.INPUT0\r\n",
      "    tensorMap:\r\n",
      "      pipeline-inputs.inputs.INPUT0: INPUT\r\n",
      "  - name: add10\r\n",
      "    inputs:\r\n",
      "    - pipeline-inputs.inputs.INPUT1\r\n",
      "    tensorMap:\r\n",
      "      pipeline-inputs.inputs.INPUT1: INPUT\r\n",
      "  output:\r\n",
      "    steps:\r\n",
      "    - mul10\r\n",
      "    - add10\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./pipelines/pipeline-inputs.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "74768602",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline load -f ./pipelines/pipeline-inputs.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "922b975e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"pipelineName\": \"pipeline-inputs\",\r\n",
      "  \"versions\": [\r\n",
      "    {\r\n",
      "      \"pipeline\": {\r\n",
      "        \"name\": \"pipeline-inputs\",\r\n",
      "        \"uid\": \"ciepgeqi8ufs73flaisg\",\r\n",
      "        \"version\": 1,\r\n",
      "        \"steps\": [\r\n",
      "          {\r\n",
      "            \"name\": \"add10\",\r\n",
      "            \"inputs\": [\r\n",
      "              \"pipeline-inputs.inputs.INPUT1\"\r\n",
      "            ],\r\n",
      "            \"tensorMap\": {\r\n",
      "              \"pipeline-inputs.inputs.INPUT1\": \"INPUT\"\r\n",
      "            }\r\n",
      "          },\r\n",
      "          {\r\n",
      "            \"name\": \"mul10\",\r\n",
      "            \"inputs\": [\r\n",
      "              \"pipeline-inputs.inputs.INPUT0\"\r\n",
      "            ],\r\n",
      "            \"tensorMap\": {\r\n",
      "              \"pipeline-inputs.inputs.INPUT0\": \"INPUT\"\r\n",
      "            }\r\n",
      "          }\r\n",
      "        ],\r\n",
      "        \"output\": {\r\n",
      "          \"steps\": [\r\n",
      "            \"mul10.outputs\",\r\n",
      "            \"add10.outputs\"\r\n",
      "          ]\r\n",
      "        },\r\n",
      "        \"kubernetesMeta\": {}\r\n",
      "      },\r\n",
      "      \"state\": {\r\n",
      "        \"pipelineVersion\": 1,\r\n",
      "        \"status\": \"PipelineReady\",\r\n",
      "        \"reason\": \"created pipeline\",\r\n",
      "        \"lastChangeTimestamp\": \"2023-06-29T14:42:04.202598715Z\",\r\n",
      "        \"modelsReady\": true\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline status pipeline-inputs -w PipelineReady | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "792680b5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"outputs\": [\r\n",
      "    {\r\n",
      "      \"name\": \"OUTPUT\",\r\n",
      "      \"datatype\": \"FP32\",\r\n",
      "      \"shape\": [\r\n",
      "        \"4\"\r\n",
      "      ],\r\n",
      "      \"contents\": {\r\n",
      "        \"fp32Contents\": [\r\n",
      "          10,\r\n",
      "          20,\r\n",
      "          30,\r\n",
      "          40\r\n",
      "        ]\r\n",
      "      }\r\n",
      "    },\r\n",
      "    {\r\n",
      "      \"name\": \"OUTPUT\",\r\n",
      "      \"datatype\": \"FP32\",\r\n",
      "      \"shape\": [\r\n",
      "        \"4\"\r\n",
      "      ],\r\n",
      "      \"contents\": {\r\n",
      "        \"fp32Contents\": [\r\n",
      "          11,\r\n",
      "          12,\r\n",
      "          13,\r\n",
      "          14\r\n",
      "        ]\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer pipeline-inputs --inference-mode grpc \\\n",
    "    '{\"model_name\":\"pipeline\",\"inputs\":[{\"name\":\"INPUT0\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]},{\"name\":\"INPUT1\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "d6fd0d45",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline unload pipeline-inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "3d5bb23c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon model unload mul10\n",
    "!seldon model unload add10"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd13b4fe",
   "metadata": {},
   "source": [
    "### Trigger Joins\n",
    "\n",
    "Shows how joins can be used for triggers as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "1a09de5c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: mul10\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/scv2/samples/triton_23-03/mul10\"\n",
      "  requirements:\n",
      "  - triton\n",
      "  - python\n",
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: add10\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/scv2/samples/triton_23-03/add10\"\n",
      "  requirements:\n",
      "  - triton\n",
      "  - python\n"
     ]
    }
   ],
   "source": [
    "!cat ./models/mul10.yaml\n",
    "!cat ./models/add10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "85eb4ef2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model load -f ./models/mul10.yaml \n",
    "!seldon model load -f ./models/add10.yaml "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "d11e29c6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model status mul10 -w ModelAvailable | jq -M .\n",
    "!seldon model status add10 -w ModelAvailable | jq -M ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78d314ab",
   "metadata": {},
   "source": [
    "Here we required tensors names `ok1` or `ok2` to exist on pipeline inputs to run the `mul10` model but require tensor `ok3` to exist on pipeline inputs to run the `add10` model. The logic on `mul10` is handled by a trigger join of `any` meaning either of these input data can exist to satisfy the trigger join."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "227e06e4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Pipeline\r\n",
      "metadata:\r\n",
      "  name: trigger-joins\r\n",
      "spec:\r\n",
      "  steps:\r\n",
      "  - name: mul10\r\n",
      "    inputs:\r\n",
      "    - trigger-joins.inputs.INPUT\r\n",
      "    triggers:\r\n",
      "    - trigger-joins.inputs.ok1\r\n",
      "    - trigger-joins.inputs.ok2\r\n",
      "    triggersJoinType: any\r\n",
      "  - name: add10\r\n",
      "    inputs:\r\n",
      "    - trigger-joins.inputs.INPUT\r\n",
      "    triggers:\r\n",
      "    - trigger-joins.inputs.ok3\r\n",
      "  output:\r\n",
      "    steps:\r\n",
      "    - mul10\r\n",
      "    - add10\r\n",
      "    stepsJoin: any\r\n"
     ]
    }
   ],
   "source": [
    "!cat ./pipelines/trigger-joins.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "2bce48e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline load -f ./pipelines/trigger-joins.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "66c60286",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"pipelineName\": \"trigger-joins\",\r\n",
      "  \"versions\": [\r\n",
      "    {\r\n",
      "      \"pipeline\": {\r\n",
      "        \"name\": \"trigger-joins\",\r\n",
      "        \"uid\": \"ciepgkqi8ufs73flait0\",\r\n",
      "        \"version\": 1,\r\n",
      "        \"steps\": [\r\n",
      "          {\r\n",
      "            \"name\": \"add10\",\r\n",
      "            \"inputs\": [\r\n",
      "              \"trigger-joins.inputs.INPUT\"\r\n",
      "            ],\r\n",
      "            \"triggers\": [\r\n",
      "              \"trigger-joins.inputs.ok3\"\r\n",
      "            ]\r\n",
      "          },\r\n",
      "          {\r\n",
      "            \"name\": \"mul10\",\r\n",
      "            \"inputs\": [\r\n",
      "              \"trigger-joins.inputs.INPUT\"\r\n",
      "            ],\r\n",
      "            \"triggers\": [\r\n",
      "              \"trigger-joins.inputs.ok1\",\r\n",
      "              \"trigger-joins.inputs.ok2\"\r\n",
      "            ],\r\n",
      "            \"triggersJoin\": \"ANY\"\r\n",
      "          }\r\n",
      "        ],\r\n",
      "        \"output\": {\r\n",
      "          \"steps\": [\r\n",
      "            \"mul10.outputs\",\r\n",
      "            \"add10.outputs\"\r\n",
      "          ],\r\n",
      "          \"stepsJoin\": \"ANY\"\r\n",
      "        },\r\n",
      "        \"kubernetesMeta\": {}\r\n",
      "      },\r\n",
      "      \"state\": {\r\n",
      "        \"pipelineVersion\": 1,\r\n",
      "        \"status\": \"PipelineReady\",\r\n",
      "        \"reason\": \"created pipeline\",\r\n",
      "        \"lastChangeTimestamp\": \"2023-06-29T14:42:27.595300698Z\",\r\n",
      "        \"modelsReady\": true\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline status trigger-joins -w PipelineReady | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "403fcbd5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"outputs\": [\r\n",
      "    {\r\n",
      "      \"name\": \"OUTPUT\",\r\n",
      "      \"datatype\": \"FP32\",\r\n",
      "      \"shape\": [\r\n",
      "        \"4\"\r\n",
      "      ],\r\n",
      "      \"contents\": {\r\n",
      "        \"fp32Contents\": [\r\n",
      "          10,\r\n",
      "          20,\r\n",
      "          30,\r\n",
      "          40\r\n",
      "        ]\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer trigger-joins --inference-mode grpc \\\n",
    "    '{\"model_name\":\"pipeline\",\"inputs\":[{\"name\":\"ok1\",\"contents\":{\"fp32_contents\":[1]},\"datatype\":\"FP32\",\"shape\":[1]},{\"name\":\"INPUT\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' | jq -M . "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "fdba4c3c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"outputs\": [\r\n",
      "    {\r\n",
      "      \"name\": \"OUTPUT\",\r\n",
      "      \"datatype\": \"FP32\",\r\n",
      "      \"shape\": [\r\n",
      "        \"4\"\r\n",
      "      ],\r\n",
      "      \"contents\": {\r\n",
      "        \"fp32Contents\": [\r\n",
      "          11,\r\n",
      "          12,\r\n",
      "          13,\r\n",
      "          14\r\n",
      "        ]\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline infer trigger-joins --inference-mode grpc \\\n",
    "    '{\"model_name\":\"pipeline\",\"inputs\":[{\"name\":\"ok3\",\"contents\":{\"fp32_contents\":[1]},\"datatype\":\"FP32\",\"shape\":[1]},{\"name\":\"INPUT\",\"contents\":{\"fp32_contents\":[1,2,3,4]},\"datatype\":\"FP32\",\"shape\":[4]}]}' | jq -M . "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "52a2bfd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline unload trigger-joins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "914d23f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon model unload mul10\n",
    "!seldon model unload add10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e017099a",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
